PATENT APPLICATION 

IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 
In re the Application of 

Thomas M. BREUEL et al. Group Art Unit: 2 1 78 

Application No.: 10/064,892 Examiner: C. PAULA 

Filed: August 27, 2002 Docket No.: 111744 

For: METHOD AND SYSTEM FOR DOCUMENT IMAGE LAYOUT 
DECONSTRUCTION AND REDISPLAY SYSTEM 

DECLARATION UNDER 37 C.F.IL Sl-131 
I, William C. Janssen Jr., a named inventor in the above-identified application, hereby 
declare and state: 

1 . This Declaration is submitted as evidence that the subject matter claimed in 
Claims 1,3-16 and 18-28 of the above-identified application was conceived by the named 
inventors and reduced to practice in the United States prior to July 13, 2001, the U.S. filing 
date of U.S. Patent Application Publication No. 2003/0014445 Al (Formanek et al.). 

2. I am a named co-inventor in the above-identified application. 

3. I have personal knowledge of the function and existence of the computer 
programs, all dated prior to July 13, 2001, that appear in the directory hsting set forth in 
Exhibit A and in the screenshots depicted in Exhibit C(l), attached to this Declaration. 

4. In the directory listing of the computer programs identified in Exhibit A and 
the screenshots depicted in Exhibit C(l), dates and other material that could indicate dates 
have been masked out, to the extent practicable, as permitted under the U.S. patent rules. 

5. The invention that was reduced to practice in the computer programs 
identified in the directory listing in Exhibit A and represented by the screenshots depicted in 
Exhibit C(l), can be summarized as follows: 
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A system and method of converting a document lin a page-image format into a form 
suitable for an arbitrarily sized display, the method comprising in sequential order the 
following steps. A document, in a page image format, is deconstructed by at least one of 
physical segmentation and logical segmentation into a set of segmented image elements 
including at least one of blocks, lines, words, characters of text, groups of characters, and 
groups of non-text characters (see Exhibit B(l) and (2); Exhibit C(2); claims 1, 3 and 4). The 
deconstructed document is then synthesized into an intermediate data structure that is 
convertible into a commercially available format (see Exhibit B(2); Exhibit C(l); claim 1). 
The synthesizing includes integrating at least one of bitmapped images, including words in 
reading order, in an intelligible display layout and links to non-textual elements (see Exhibits 
B(l) and (2); Exhibits C(l) and (2); claims 6 and 7). The synthesizing also includes 
converting non-text image areas, layout properties and segmented image areas, into the 
intermediate data structure (see Exhibits B(l) and (2); Exhibits C(l) and (2); claim 5). The 
intermediate data structure is (1) capable of being stored in a storage device; (2) adaptable to 
at least one of display screen size, page size, resolution, contrast, color and geometry, at the 
time of display; and (3) adaptability supported by at least one of repagination of text, 
reflowing of text, logical links of text to associated text and non-textual content (see Exhibits 
S(l)-(4); Exhibits C(l) and (3); claims 8, 12 and 13). The intermediate data structure is then 
distilled using commercially avedlable software for redisplay by converting the intermediate 
data structure into a format usable for an arbitrarily sized display (see Exhibits B(2)-(4); 
Exhibits C(l) and (3); claim I). This includes redisplaying the document in hximan readable 
format to include (1) at least one of an electronic book format, Internet browsable format and 
a print format; or (2) a device specific display format (see Exhibits B(3) and (4); Exhibit 
C(3); claims 9-1 1). The intermediate data structure is automatically adaptable at the time of 
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display to constraints of display devices or circumstances of viewing (see Exhibit B(2); 
Exhibit C(l); claim 1). 

6. The invention that was reduced to practice in the computer programs 
identified in the directory listing in Exhibit A was conceived prior to July 13, 2001. This 
invention is claimed in the above-identified application. 

7- On August 9, 2006, the inventions represented by the programs listed in 
Exhibit A were used to produce the intermediate format listed in Exhibit B(2)and thereby 
show the reduction to practice of a system and method of converting a document in a page- 
image format into a form suitable for an arbitrarily sized display as described in paragraph 6 
herein. Exhibit B(l) illustrates an electronically stored document (1) in a page-image format. 
Exhibit B(2) illustrates an intermediate data format (2) of the electronically stored file of 
document (1). Exhibit B(3) illustrates a dynamic reflow of the intermediate data format (2) 
into a first arbitrarily sized display. Exhibit B(4) illustrates a dynamic reflow of the 
intermediate data format (2) into a second arbitrarily sized display. 

8. On or about March 2, 2007, programs listed in Exhibit A and represented in 
the screenshots depicted in Exhibit C(l), were applied to a document image, represented by 
Exhibit C(2). The progression of the applications is represented in Exhibit C(l). The first 
program, BatchSegmenter^ identifies individual word images and their reading order, and 
writes out an analysis of the document giving that information. The second program, 
chop.py, reads this analysis, and creates a version of the document in intermediate data 
format. Exhibit C(3) shows a reflowed page of the intermediate format viewed in an HTML 
browser. 

9. I hereby declare that all statements made herein of my own knowledge are 
true, and that all statements made on information and belief are believed to be true; and 
further that these statements were made with the knowledge that willful false statements and 
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the like so made are punishable by fine and/or imprisonment under Section 1001 of Title 18 
of the United States Code, and that such willful false statements may jeopardize the validity 
of the application or any patent issuing therefrom. 

t WilR^C. Janssen 

Attachments: / 




Exhibit A 
Exhibits B(1H4) 
Exhibits C(l)-(3) 



Exhibit A 



% la -1 /project/cuda/janssea/segl/chop.py /project/did/erc/util/seg 
-rwxr-xr-x 1 tbreuel pare 80 Mar 11 [^| BatchSegmenter . ah 



-rwcr-xr-x 1 tbreuel pare 76 Mar 11 Ref lowHtml . ah 

drwxr-xr-x 3 tbreuel pare 25032 Feb 17 data 

-rw-r--r-- 1 tbreuel pare 163242 Mar 11 aeg.jar 

-rwxr-xr-x 1 tbreuel pare 1543 Feb 19 H ahow-rect 
% 
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Real-Time Control of a Virtual Human Using Minimal Sensors 

- - Norman L~Badler 

Midiael J. HoDick 
John P. Gianieri 



Computer Gra.pluca Researdi Laboratory 
Department of Computer and Information Science 
Uniyeraity of PennsyiTania 
Philadelphia, PennayWania 19104-6389 

1 Abstract 

We track, ts resl-iime, the poGition and paatnxe of a 
}.«>nj.n body^ uoins n ixuniizial zniZDber of 6 DOF oea- 
ooxs to captuze fall body atanding pootures. We iu» 
4 oenoon to create a ffood apprfnimatinn of a htunaa 
operafcor'o poutlon and pootuze, and map it on to oor 
articuUted con^utet grapbico buxnaa modeL The 
„mrnff^ Joiato are pooUioned by a tbot inreroe klne- 
maticD BlgariUun. Our goal in to lealinticaliy recreate 
buaum pootureo whQe nbdinany cncninbeKias the op- 
erator. 



2 Background 

UeaOy, o virtual euTiroument interface ehould be able 
to neasnre sad recreate a partidpaak*! pootnre exact- 
ly. Bather tbaa the tradiSoaal -^cSscmbodied baad* 
appreacb, vn mid lOte to geuerate a complete, real- 
irtkal^^pdotared Enixnaa ImBga. Kcrsnm, the iqnTp-' 
meuk needed to accnzately track every body oegmeat 
(or joint angle) ofabunwa Id costly and eumbemoma. 
We Eoce orveral questioBK hem closely nnist the vSr- 
tuol hunuB^D poftuxe match the opemtor*ii, oad hov 
nxuch infbnaatiaa do we need to achieve tUo degree 
of realiBzn? 

Ab ia other areaa of VR, the degree of realiain nee- 
esoazy depcado greatly on the taako we would Uha 
to perfbzm [11]. We have concen t rated on creating 
an intetfoec that will allocr a buman participant to 
pofonn bainc taska, uaiBg a niunbex of sen- 

aoro to derive feainble, rrnnonahly occuxatc pooturen. 
Three pieces of infimnation axe esoential for our pos- 
ture recenatmctioa algottthm: the partidpant'SB vieir 
vector, center of man, aad the locatioa of the cnd- 
efbcton the partidpant will uk to interact with the 



Figure 1: Seaoor Placement and Support Polygon 
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Real - Time Control of a Virtual 

-HximanUsing Miniraai Sensors- 
Norman L Badler Mi chae^l J. Hollick John R 
Granieri Computer Graphics Research 
Laboratory Department oi Computer and 
Information Science Umversity of 
Pennsylvania Philadelphia, PennBylvania 
191 04 - 6389 1 Abstract W e track, m real - 
time, the position and posture of a human body, 
uBmg a Tninimal nuidber of 6 D OF sen sors fco 
capture f uU body standing postures. W e use 4 
sensors to create a Rood approximation of a 
hiunan operator a position and posture, and map it 
on to our articulated computer ^ aphics human 
model. The uxisensed joints are positioned by a f 
ast mvierse kine - matics algorithm. Our goal la to 
realistically recreate human postuixes while 
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3.2 Multi-Document Reading 

Because document finding is typically separated from docimient reading, m\il- 
tiple document reading tasks are particularly difficult in the electronic realm. 
Many applications provide facilities for navigating between multiple documents 
of the same type, such as tabs In a web browser. However, reading tasks may 
involve moving between several types of documents such as web pages, word 
processing documents, presentation slides, etc. This type of task suggests the 
need for techniques to transfer attention rapidly among documents, regardless 
of type, in small working sets. 

A related problem is being able to view the pages of multiple documents 
at once. With paper, this is often accomplished by spreading out documents 
on a table. Reading applications often provide thumbnail facilities for viewing 
multiple pages from a single document but not from multiple documents. The 
ability to do so can be important for making comparisons [15] and for getting 
an overview of a collection of documents [16]. 

3.3 Finding Documents 

Perhaps the most widely accepted method of finding relevant documents is key- 
word search. This method has a number of limitations in the case of personal 
digital libraries. Keyword search is fundamentally a recall-based user interface, 
A classic principle in user interface design is to minimize the memory load on the 
liser [17]. This is often referred to informally as "supporting recognition rather 
than recall." 

A general problem with keyword search is that a user's vocabulary often does 
not match the desired document's vocabulary. This means that users may not 
be able to find certain docimients or they may have to experiment with several 
queries before they find what they want. A number of techniques have been 
introduced to deal with this problem such as term aliasing or personalizations 
like those in Haystack [18]. Nevertheless, these heuristics are not likely to work 
in all cases. It is also often the case that the user is not able to formulate a query 
at all. Some examples include searching for a picture, a specific page layout, 
or some other visual property of a document. In these cases, the user needs 
techniques for interacting with documents visually rather than textually. 

4 The 17C System 

The UC system developed in our research group integrates a number of recent 
user interface, information visualization, and digital library techniques with the 
goal of addressing the problems described in the foregoing. Tlie system is also 
designed to work well with pen-based tablet computers where traditional user 
interface controls, such as scrollbars and text boxes, are of limited effectiveness. 
While the foregoing sections have described the motivation and design rationale 
for this system, the present section and the next describe the system in detail. 
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IN THE UNITED STAreS PAIEOT Al^ TRADEMARK OmCE 



PATFNT APFT.Tr ATfON 



In re the Application of 



Thomas M. BREUEL et al. 



Group Art Unit: 2178 



Application No.: 10/064,892 



Exanuner: 



C. PAULA 



Filed: 



August 27, 2002 



Docket No.: 111744 



For: 



METHOD AND SYSTEM FOR IXX:UMENT IMAGE LAYOUT 
DECONSTRUCTION AND REDISPLAY SYSTEM 



nFrT ARATTON TTNDF.R 37 C.YJL SL131 



I, Thomas M. BREUEL, a named inventor in the above-identified application, her^y 
declare and state: 

1 . This Declaration is submitted as evidence that the subject matter claimed in 
Claims 1, 3-16 and 18-28 of the above-identified application was conceived by the named 
inventors and reduced to practice in the United States prior to July 13, 2001, the U.S. filing 
date of U.S. Patrat Application Publication No. 2003/0014445 Al (Fonnanek et al.). 

2. I am a named co-inventor in the above-identified application. 

3. I have reviewed the attached Declaration of William C. Janssen Jr., another 
named co-inventor in the above-idratified application, and confirm that, based on 
information and belief, the assertions made in Mr. Janssen's Declaration are true. 

4. I hereby declare that all statements made herein of my own knoT^edge are 
true, and that all statements made on information and belief are believed to be true; and 
fijrther that these statements were made with the knowledge that willfiil false statements and 
the like so made are punishable by fine and/or imprisonment under Section 1001 of Title 18 
of the United States Code, and that such willfiil false statements may jeopardize the validity 



\ Application No. 10/064,892 

of Ifae application or any patent issuing therefrom. 



Date: 

i 7 liiomasMBREUEL 

Attachments: 

Declaration of ^iliam C. Janssen Jr. dated March 20, 2007 




PATENT AP PLICATIOM 

IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 
In re the Appflcabon of 
Thomas M. BREUEL et al. 
Application No.: 10/064,892 
Fifed: August 27, 2002 



Group Art Unit 2178 
Examiner: C. PAULA 
Docket No.:iai744 



DECLARATION UMDPB ^7 p T ?T m 

I, Henry S. BAIRD, a named Inventor in the above-identified 
application, hereby declare and state: 

1. This Declaration is submitted as evidence that the subject 
matter claimed In Claims 1, 3-I6 and 18-28 of the above-Identified 
application was concehred by the named Inventors and reduced to 
practice In the United states prior to July 13, 2001, the U.S. filing date of 
U.S. Patent Application Publication No. 2003/0014445 AI (Fonnanek et al.). 

2. lama named co-inventor in the above-identified application. 

3. I have reviewed the attached Declaration of William C. 
Janssen jr.. another named co-Inventor in the above-Identified application 
and confirm that based on information and belief, the assertions made In' 
Mr. Janssen's Decia ration are true. 

4. I hereby declare that all statements made herein of my own 
knowledge are true, and that all statements made on Infcnnatfon and 
belief are believed to be true; and further that these statement were 
made with the knowledge that willful feise statements, and the lilce so " 



made are punishable by fine and/or imprisonment under Section 1001 of 
Title 18 of tlie United States Code, and that such wfllfxil false statements 
may jeopardize the validity of the application or any patent issuing 
ttierefirom. 
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PATENT AFFOCATION 

IN THE UNITED STATES PATENT AND TRADEMARK OFHCE 
In re tbe Application of 

Thomas M. BREUEL et al. Group Art Unit: 2178 

Application No.: 10/064,892 Examiner C.PAULA 

Filed: August 27, 2002 Docket No.: 111744 

For METHOD AND SYSTEM FOR DOCUMENT IMAGE LAYOUT 
DECONSTRUCTION AND REDISPLAY SYS11EM 

DECLARATION UNDER 37 C.F.R, Sl,131 

I, Ashok C. POP AT, a named inventor in the above-identified application, hereby 
declare and state: 

1 . This Declaration is submitted as evidence that the subject matter claimed in 
Clauns 1, 3-16 and 18-28 of the above-identified application was conceived by the named 
inventors and reduced to practice in the United States prior to July 13, 2001, the U.S. filing 
date of U.S. Patent Application Publication No. 2003/0014445 Al (Fonnanek et al.). 

2. I am a named co-inventor in the above-identified application. 

3. I have reviewed the attached Declaration of William C. Janssen Jr., another 
named co-inventor in ttie above-identified application, and confim that, based on information 
and belief; the assertions made in Mr. Janssra's Declaration are true. 

4. I hereby declare that all statements made herein of my own knowledge are 
true, and Aat all statements made on information and belief are believed to be trae; and 
further that these statements were made with the knowledge that willful false statements, and 
the like, so made are punishable by fine and/or imprisonment under Section 1001 of Title 1 8 




Application No. 1 0/064,892 

of the United States Code, and that such willfixl false statements may jeopardize the validity 
of the application or any patent issuing therefinom. 

AshokC. POPAT 

Attachments: 

Declaration of William C. Janssen Jr. dated March 20, 2007 
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